Managing a Critical Situation with a Team
written by Umberto Ottavianelli
In full health emergency and just before December 31, 2020, I had to manage the ISP/SP exchange migration of a renowned hospital in Italy. From these few lines, we can already guess the complexity of the situation due to the very nature of the customer indicated. The real challenge was to make sure that the disruption was reduced to a minimum. For this reason, I decided to take action before the current SP stopped providing the service. In other words, I made sure that the current SP and the one to which it migrated (this is the ISP/SP where I work) coexisted.
The first activity was carried out on the CED side (central office) where two redundant routers were configured, one master and the other slave, and was interfaced with the customer firewall sharing the same private IP addressing plan of their current ISP/SP. Subsequently, the branch offices were managed on the MPLS network with 100Mb fiber optic technology. In particular, I only managed the configuration of one of these locations and I wrote a template uploaded to the cloud of my ISP/SP to ensure that delivery colleagues implement the same configuration on the remaining peripheral locations.
The migration of these offices was expected in a second step just before Christmas day. During the migration, the customer informed us that he was in a disruption with the VoIP phones. Unfortunately, until that moment, there had never been talk of this service as the customer believed it was in his management and not managed by the current ISP/SP. In this regard, in order not to have to do rollback, the LAN was created to manage VoIP (also on the CED side) and management QoS was implemented both in the router and in the circuit. The customer seemed satisfied so we were ready for the migration of the next locations.
After a few days, just before December 31, 2020, the customer experiences problems when the peripheral office that is with my ISP/SP has to communicate with the ISP/SP currently used. After careful analysis, it was assumed that the VoIP traffic, on the head office side, passed through a firewall that the customer believed he had managed properly. At that point, a new firewall was installed and configured by the SOC colleagues so as to allow only the VoIP LAN. Once the ethernet cables were properly connected, VoIP also started to work properly with great customer satisfaction.
At that point, the hospital IT chief provided us with the current VoIP IP addressing plan used across all sites. Thanks to this data, I modified the template that I uploaded to the cloud (including updated data; these data are indexed by location and when the template matches the reference site, it correctly extrapolates the data). At that point, the update of the network device configurations of the peripheral offices was managed in delivery and the entire network infrastructure was successfully migrated.
In the end, the client wanted to thank me by calling me personally on the office phone! He understood that the challenge was anything but trivial but that the commitment was maximum. After a few days, he sent an e-mail to my manager to congratulate me on my work and on the other people who, together with me, have committed themselves to building the entire network infrastructure. It was a great satisfaction to have completed this job without giving any disservice and with the compliments of the customer.
About the author:
Hi everyone, my name is Umberto Ottavianelli and I am a Senior Network Engineer. I collaborate with the most important Italian ISP and I deal with the design and implementation of CED offices on multivendor technology. I have always been passionate about technology and fascinated by the world of cybersecurity that I study with and for passion and, often, I find it useful when I interact with Firewall specialists or with the SOC guys.